About upgrading to 9.3 READ THIS FIRST

Read about changes in behavior to note for Splunk Enterprise and Splunk Universal Forwarder when you upgrade to version 9.3.

Lower or higher versions of this topic might present different information than the information for your target version. Use the Version drop-down list to choose the product version to which you're upgrading.

Administrators and advanced users can review changes to the .conf and .conf.spec files by using the Compare feature on different branches of the files in the jewnix/splunk-spec-files GitHub repository. Splunk thanks David Twersky (jewnix) for providing and maintaining this repository, and links to its contents with his permission.

For general instructions on how to upgrade Splunk Enterprise, see How to upgrade Splunk Enterprise.

Splunk App and Add-on Compatibility

Not all Splunk apps and add-ons are compatible with Splunk Enterprise version 9.3.

• See the Splunk products version compatibility matrix for information about which versions of Splunk IT Service Intelligence and Splunk Enterprise Security are compatible with this version of Splunk Enterprise.
• You can visit Splunkbase to confirm that your apps and add-ons are compatible with this version.

If your app or add-on is not compatible with version 9.3, consider delaying your upgrade until a compatible version is available.

Key points for upgrading to version 9.3

The following is a list of important items that you must consider before you upgrade Splunk Enterprise and its components. The sections that follow provide supporting details for these key points. Read through all sections in the topic before you begin your upgrade activities.

• In Splunk Enterprise 9.3, usage of older jQuery libraries is restricted by default. Admins can still enable the use of older jQuery libraries in a self-serviceable manner. See Older jQuery libraries are restricted by default.
• Use the Splunk Products version compatibility matrix to ensure that any premium Splunk apps and add-ons you run are compatible with version 9.3. If they are not, do not upgrade until compatible versions become available.
• See How to upgrade Splunk Enterprise for information on supported upgrade paths to Splunk Enterprise 9.3.
• Splunk supports a direct upgrade of a universal forwarder to version 9.3 from versions 8.2.x and higher of the universal forwarder..
• To upgrade search head or indexer clusters, see Follow specific instructions to upgrade clusters later in this topic.
• Back up all App Key Value Store (KV store) databases prior to starting an upgrade. For information about backing up your KV store, see Back up and restore KV store in the Admin Manual.
• If you run Linux machines that use the second extended (ext2) file system, upgrade that file system to third extended (ext3) prior to starting an upgrade.
• Python 3.9 is the default interpreter. Please ensure that all apps and add-ons are on the latest version and compatible with Python 3.9, otherwise there may be breakage.

Changes that can potentially break Splunk Enterprise installations

There is no supported method for rolling back an installation to a prior release. Follow the guidance for these critical items to avoid breaking your existing installation during an upgrade.

The log severity level for searches with infix wildcards has increased from INFO to WARN

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.3

Certain searches that produce inconsistent search results now display the following message as a warning instead of an info message:

The term <term> contains a wildcard in the middle of a word or string. This might cause inconsistent results if the characters that the wildcard represents include punctuation. Learn More

.

If you don't want Splunk Enterprise to log this message as a warning, you can reduce the log severity level back to INFO. To change the message to an INFO-level message, follow these steps:

1. Open or create a local messages.conf configuration file at $SPLUNK_HOME/etc/system/local if you are using *nix, or %SPLUNK_HOME%\etc\system\local if you are using Windows. For information about how to edit .conf files, see How to edit a configuration file in the Splunk Enterprise Admin Manual.
2. Edit the severity level for [UNIFIEDSEARCH:SEARCH_CONTAINS_INFIX_WILDCARD__S].
3. Save the file and close it.

Role-based field filters have been replaced with field filters

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.3
If you used the role-based field filters preview feature in a previous release of Splunk Enterprise, you must now use field filters instead to protect your sensitive data. Role-based field filters have been replaced with field filters. For more information about field filters, see Protect PII, PHI, and other sensitive data with field filters.

READ THIS FIRST: Should you deploy field filters in your organization?

Field filters is a powerful tool that can help many organizations protect their sensitive fields from prying eyes, but it might not be a good fit for everyone. If your organization runs Splunk Enterprise Security or if your users rely heavily on commands that field filters restricts by default (mpreview, mstats, tstats, typeahead, and walklex), do not use field filters in production until you have thoroughly planned how you will work around these restricted commands. See READ THIS: Restricted commands do not work in searches on indexes with field filters in the Securing Splunk platform manual.

Upgrade from role-based field filtering to field filters

If you used role-based field filtering in a previous preview release and are concerned about a gap in protection for your sensitive data while you're upgrading to this release and subsequently transitioning to field filters, follow these instructions:

Before upgrading to this release

1. Manually back up settings for your old role-based field filters that are stored in the authorize.conf file.
2. Restrict access to the protected indexes by assigning the srchIndexesAllowed capability or the srchIndexesDisallowed capability to roles in Splunk Web in the Roles page in Settings.

After upgrading to this release

1. Create new field filters in Splunk Web.
2. Manually remove settings for your old role-based field filters that are stored in the authorize.conf file.
3. Restore access to the protected indexes by unassigning the srchIndexesAllowed capability or the srchIndexesDisallowed capability from roles in Splunk Web in the Roles page in Settings.

Default value change for the 'max_documents_per_batch_save' setting causes restore from KV store backups made using versions earlier than Splunk Enterprise 9.3.0 to fail

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.3

In Splunk Enterprise 9.1.x and 9.2.x, the default value for the max_documents_per_batch_save setting in the [kvstore] stanza of the limits.conf.in file is 50000. However, in Splunk Enterprise version 9.3.0, the default value for the max_documents_per_batch_save setting changed to 1000.

As a result of this change in the default value for the max_documents_per_batch_save setting, you must restore KV backups made using Splunk Enterprise 9.2.2 and earlier versions before upgrading to Splunk Enterprise version 9.3.0. If you try to restore a KV backup in Splunk Enterprise version 9.3.0, your restore will fail and Splunk software displays a message indicating that there are too many documents for a single batch save.

The Windows Event Log data input has changes in how it expects incoming events from Windows Event Collector subscriptions

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1

The Windows Event Log input now has a setting that lets you specify the format of incoming events from a Windows Event Collector (WEC) subscription. Previously, the input chose the event format based on the name of the destination log for the WEC subscription. Now, you can specify the format that the Splunk platform is to expect when it receives events through the subscription.

The name of the destination log for the WEC subscription determines the default format that the Splunk platform expects for incoming events from a subscription. For subscriptions whose destination log names begin with the string ForwardedEvents, the event format is rendered_event, or "RenderedText" in Microsoft parlance. For all other destination log names, the event format is raw_event, or "Events" in MS parlance.

If you have WEC subscriptions whose destination log names do not begin with ForwardedEvents but want to use the RenderedText format for that subscription, set wec_event_format to a value of rendered_text in the inputs.conf configuration file stanza for the subscription whose event format you want to change. See the following example:

[WinEventLog://WEC-Channel1]
wec_event_format = rendered_event

For more information about the setting, see Monitor Windows Event Log data in the Getting Data In Manual.

Splunk Enterprise 9.1 fixes a critical vulnerability in deployment server but might introduce problems for older deployment clients

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

Splunk introduced a fix for a vulnerability in deployment server in version 9.0 of Splunk Enterprise, and this fix is also available in version 9.1. If you run a deployment server, upgrade that server to version 9.1 of Splunk Enterprise as soon as possible. Before the upgrade, carefully review your deployment server setup and the current versions of the deployment clients in your Splunk Enterprise network. Depending on the setup of your deployment server and whether that component shares a computer with other Splunk Enterprise components, you might need to do the following to ensure your deployment server and clients communicate without problems:

• Isolate deployment server from other components on a machine. Isolating your deployment server means you only have to upgrade that component. The sole exception for isolation is if you run a deployment server and a license manager on the same machine.
• Confirm that all deployment clients in your network run version 7.0.0 or higher of Splunk Enterprise or the universal forwarder. You don't have to upgrade deployment clients to version 9.0.0, but they must be at version 7.0.0 or higher to communicate with version 9.0.0 deployment servers.

See the following topics for additional information:

• SVD-2022-0608 for more about the vulnerability.
• Security updates in the Securing Splunk Enterprise Manual for more about the security updates that come with Splunk Enterprise 9.0 and higher.
• Client version compatibility in the Updating Splunk Enterprise Instances Manual for more about which versions of deployment client that the version 9.0.0 deployment server supports.

The Linux universal forwarder installer creates a new non-root operating system user called 'splunkfwd'

Applicable components: Universal forwarder
Applicable OSes: Linux
Version introduced: 9.0

If you install version 9.1 or higher of the Linux universal forwarder, the installer now creates the 'splunkfwd' operating system user as a non-root user for managing your universal forwarder. In previous versions, the default user is 'splunk'. The 'splunkfwd' user is a new "least privileged" user that provides only the capabilities necessary to run the universal forwarder, which improves security. To learn more about how to work with the 'splunkfwd' user, see Manage a Linux least-privileged user in the Universal Forwarder manual.

Follow specific instructions to upgrade clusters

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 7.3

To upgrade indexer or search head clusters, follow the upgrade procedure for the type of deployment you have.

• If your deployment has indexer clusters, follow the index cluster upgrade instructions.
• If your deployment has search head clusters, follow the search head cluster upgrade instructions.

On Splunk Enterprise indexer cluster manager nodes, enable maintenance mode before you upgrade

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 8.1

Because of changes to bucket data with the release of the latest version of Metrics Store, indexers that run versions of Splunk Enterprise lower than 8.1 cannot handle bucket replications from versions that run 8.1 and higher. Before you upgrade a cluster manager node, ensure it is in maintenance mode first. This is a standard part of the indexer cluster update process.

See Use maintenance mode in Managing Indexers and Clusters of Indexers.

Back up App Key Value Store prior to starting an upgrade

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 7.3

Back up the app key value store (KV store) before any maintenance like an upgrade.

Confirm that you have accounted for this downtime in your upgrade planning. See Back up and restore KV store for more information.

The indexer cluster slave-apps directory is renamed to peer-apps

Applicable components: Splunk Enterprise
Applicable OSes: all

During upgrade of a Splunk Enterprise indexer cluster, the installer renames the slave-apps directory on an indexer cluster node to peer-apps. To accommodate this renaming, you must manually change any external hard-coded references to slave-apps, such as in external scripts, TLS certificates, and so on, to peer-apps. See Update common peer configurations and apps in Managing Indexers and Clusters of Indexers.

The setting master_uri is renamed to manager_uri

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

In the [License] stanza of the server.conf configuration file, the master_uri setting has been renamed to manager_uri. The old setting name is deprecated but still usable in 9.2. During an upgrade to 9.x, if you have a mixed deployment of 9.x and 8.x instances, do not change master_uri to manager_uri in the 8.x instances. Version 8.x instances do not recognize the manager_uri setting name.

Occurrences that appear to be problems but are not

You might see things happen during or immediately after an upgrade that appear to indicate that the upgrade is not working. In nearly all cases, the occurrences that happen here can be expected.

If the following things occur during the upgrade, let the upgrade continue and do not interrupt it. If they occur immediately afterward, then perform benchmark tests on the deployment and compare to any benchmarks that you set up as part of the first phase of upgrading. Consider involving Splunk Support only if those benchmarks differ by a significant margin.

• On indexers, memory and CPU usage increases due to the following:
  • New data ingestion pipelines
• Permissions on the Splunk Enterprise introspection directory might change. Confirm that the user that runs Splunk Enterprise has write permission to the $SPLUNK_HOME/var/log/introspection directory.
• Deployment servers might push updates to all deployment clients due to app bundle hash recalculations.

The Splunk platform now warns you when there is no configuration for allowed internet domains for alert actions that send emails

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.3

To improve security posture for Splunk Enterprise environments, Splunk implemented a change to how it processes the sending of emails as a result of an alert action. The allowedDomainList setting in the alert-actions.conf configuration file controls which internet domains to which the instance can send alert action emails. If the setting doesn't have a value, then the instance can send email to any domain as part of an alert action. This is a potential security risk.

If you have not configured an allowed email domain list with the allowedDomainList setting, then the Splunk Enterprise instance advises you of that exposure with the following message:

Security risk warning: Found an empty value for 'allowedDomainList' in the alert_actions.conf configuration file. If you do not configure this setting, then users can send email alerts with search results to any domain. You can add values for 'allowedDomainList' either in the alert_actions.conf file or in Server Settings > Email Settings > Email Domains in Splunk Web

This message appears by design on any type of search head. It is a security posture that you, as a customer, are responsible for maintaining to reduce the potential of spam messaging and data exfiltration. To configure a list of valid internet domains to which your search heads can send email, see Configure email notification for your Splunk instance in the Alerting Manual.

If the message appears on an indexer, you can safely dismiss the warning without further action. Consider configuring an internet domain that does not send email and using that domain for this configuration to suppress further warnings.

Changes to deployment server architecture

Applicable components: Splunk Enterprise and Splunk Universal Forwarder
Applicable OSes: all
Version introduced: 9.2

See Upgrade pre-9.2 deployment servers in Updating Splunk Enterprise Instances for important details related to these changes.

Significant aspects of the deployment server have been redesigned in Splunk Enterprise version 9.2 to improve performance and manageability. Because of these architectural improvements, deployment server upgrades that span the version 9.2 release automatically undergo changes to implement these improvements.

Considerations for changed or removed features

The following major features have been changed or removed from this version. If you use features that have been removed, and have not yet migrated off them, consider delaying your upgrade until you have.

Changes to login_content setting in web.conf

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1.5, 9.2.2, 9.3

For security purposes, the login_content setting in web.conf no longer executes scripts.

A new capability has been added that lets you edit passwords stored within an app

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1

A new capability has been added that lets you edit credentials that have been stored within a Splunk app. This capability, edit_storage_passwords, can be assigned to a role to grant the ability to make changes to passwords stored within the context of an app. Previously, you needed the admin_all_objects capability to make these kinds of changes. While the admin_all_objects capability still lets you modify stored passwords in an app, it is no longer a requirement and lets you further secure your environment by reducing the number of roles that need the superuser-level capability.

After you upgrade, review any cloned roles that include the admin_all_objectscapability in the context of creating and updating passwords in an app. Consider replacing that capability with the new edit_storage_passwords capability instead.for these roles. You can also use the REST API to modify the access control list of each app to apply role restrictions to the apps' password storage configuration, including but not limited to restricting the ability to change passwords within an app to a specific role.

After you perform these adjustments, users who hold roles that contain the edit_storage_passwords capability can only modify credentials in apps to which their roles specifically grant them access. For example, if a user holds a role that grants them access to writing passwords in app1, they can't write passwords in app2 unless they hold a role that contains either or both of the edit_storage_passwords or admin_all_objects capabilities. The admin_all_objects capability overrides the edit_storage_passwords capability and still gives users the ability to see and change storage passwords on any Splunk app.

For more information about app access control endpoints (ACLs) and the REST API, see Access endpoint desecriptions in the REST API Reference Manual.

-->

Older jQuery libraries are restricted by default

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1

Usage of older jQuery libraries is restricted by default, but admins can still enable the use of older jQuery libraries in a self-serviceable manner. Select Settings, then Server Settings. Then, from the Internal Library Settings page, toggle the jQuery Libraries older than 3.5 section.

To learn more about enabling and restricting access to older jQuery libraries, see Control access to jQuery and other internal libraries.

Index buckets use zstd compression by default

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The indexes.conf configuration file setting journalCompression has a default of "zstd" beginning with this release.

The default tsidxWritingLevel for indexes has changed

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The default tsidxWritingLevel in indexes.conf has changed from 2 to 3, beginning with this release.

Splunk Enterprise 7.2 introduced a new file format and optimizations for tsidx files that resulted in improved search performance through decreased I/O, lowered storage usage, and improved utilization of SmartStore caches. These optimizations are encapsulated in levels, with new levels added in higher releases of Splunk Enterprise. Changing the default tsidxWritingLevel changes the optimizations used by both the index tsidx files and data model accelerations. See The tsidx writing level in the Managing Indexers and Clusters of Indexers manual.

The IOWait status check is delayed on startup

Applicable components: Splunk Enterprise
Applicable OSes: Linux
Version introduced: 9.0

The IOWait Health Report incorporates a 120 second delay after the Splunk Enterprise service starts to prevent false alerts.

The Bucket Health alert threshold is more aggressive

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The threshold values for the Health Report "Bucket" feature are set to alert more aggressively when an indexer rolls a large number of small buckets.

The web.conf setting 'simplexml_dashboard_create_version' is deprecated

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.0

The simplexml_dashboard_create_version setting in the web.conf configuration file has been deprecated. Changing the default value might introduce security risks. Do not change the value without consulting Splunk Support.

Considerations for new features

Splunk has introduced the following new features in this version of Splunk Enterprise. You might need to perform some configuration after an upgrade to enable and take advantage of these features.

Macros now replicate by default to search peers

Applicable components: Splunk Enterprise
Applicable OSes: all
Version introduced: 9.1

Macros used in apps are now replicated by default to search peers as part of the knowledge bundle in Splunk deployments. As a result of this change, searches that previously failed now run successfully, which could affect downstream performance.

If you don't want to replicate macros for your apps, you can suppress replication by setting replicate.macros = false in the [replicationSettings:refineConf] stanza in the distsearch.conf file. Be aware that disabling distribution of macros might negatively impact your search results.

Learn about known upgrade issues

To learn about any additional upgrade issues for Splunk Enterprise, see the Known Issues - Upgrade Issues page in the Release Notes.